AutoML Leaderboard
| Best model | name | model_type | metric_type | metric_value | train_time |
|---|---|---|---|---|---|
| 1_DecisionTree | Decision Tree | logloss | 0.377679 | 7.8 | |
| 2_Default_LightGBM | LightGBM | logloss | 0.102785 | 4.36 | |
| 3_Default_Xgboost | Xgboost | logloss | 0.0883173 | 2.47 | |
| 4_Default_CatBoost | CatBoost | logloss | 0.0759057 | 2.99 | |
| 5_Default_RandomForest | Random Forest | logloss | 0.234308 | 3.65 | |
| 6_Default_ExtraTrees | Extra Trees | logloss | 0.363181 | 3.23 | |
| the best | Ensemble | Ensemble | logloss | 0.074684 | 0.74 |
AutoML Performance
AutoML Performance Boxplot
Features Importance
Spearman Correlation of Models
Summary of Ensemble
Ensemble structure
| Model | Weight |
|---|---|
| 3_Default_Xgboost | 1 |
| 4_Default_CatBoost | 4 |
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.074684 | nan |
| auc | 0.996244 | nan |
| f1 | 0.980344 | 0.622596 |
| accuracy | 0.974803 | 0.622596 |
| precision | 1 | 0.99416 |
| recall | 1 | 0.000136718 |
| mcc | 0.945262 | 0.622596 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.074684 | nan |
| auc | 0.996244 | nan |
| f1 | 0.980344 | 0.622596 |
| accuracy | 0.974803 | 0.622596 |
| precision | 0.98155 | 0.622596 |
| recall | 0.979141 | 0.622596 |
| mcc | 0.945262 | 0.622596 |
Confusion matrix (at threshold=0.622596)
| Predicted as CONFIRMED | Predicted as FALSE POSITIVE | |
|---|---|---|
| Labeled as CONFIRMED | 440 | 15 |
| Labeled as FALSE POSITIVE | 17 | 798 |
Learning curves
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 2_Default_LightGBM
LightGBM
- n_jobs: -1
- objective: binary
- num_leaves: 63
- learning_rate: 0.05
- feature_fraction: 0.9
- bagging_fraction: 0.9
- min_data_in_leaf: 10
- metric: binary_logloss
- custom_eval_metric_name: None
- explain_level: 2
Validation
- validation_type: split
- train_ratio: 0.75
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
3.7 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.102785 | nan |
| auc | 0.993536 | nan |
| f1 | 0.978049 | 0.478538 |
| accuracy | 0.971654 | 0.478538 |
| precision | 1 | 0.997603 |
| recall | 1 | 0.00187858 |
| mcc | 0.938192 | 0.478538 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.102785 | nan |
| auc | 0.993536 | nan |
| f1 | 0.978049 | 0.478538 |
| accuracy | 0.971654 | 0.478538 |
| precision | 0.972121 | 0.478538 |
| recall | 0.984049 | 0.478538 |
| mcc | 0.938192 | 0.478538 |
Confusion matrix (at threshold=0.478538)
| Predicted as CONFIRMED | Predicted as FALSE POSITIVE | |
|---|---|---|
| Labeled as CONFIRMED | 432 | 23 |
| Labeled as FALSE POSITIVE | 13 | 802 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 6_Default_ExtraTrees
Extra Trees Classifier (Extra Trees)
- n_jobs: -1
- criterion: gini
- max_features: 0.9
- min_samples_split: 30
- max_depth: 4
- eval_metric_name: logloss
- explain_level: 2
Validation
- validation_type: split
- train_ratio: 0.75
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
2.5 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.363181 | nan |
| auc | 0.945952 | nan |
| f1 | 0.893462 | 0.564351 |
| accuracy | 0.861417 | 0.564351 |
| precision | 1 | 0.782273 |
| recall | 1 | 0.103251 |
| mcc | 0.71898 | 0.692002 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.363181 | nan |
| auc | 0.945952 | nan |
| f1 | 0.893462 | 0.564351 |
| accuracy | 0.861417 | 0.564351 |
| precision | 0.88172 | 0.564351 |
| recall | 0.905521 | 0.564351 |
| mcc | 0.695869 | 0.564351 |
Confusion matrix (at threshold=0.564351)
| Predicted as CONFIRMED | Predicted as FALSE POSITIVE | |
|---|---|---|
| Labeled as CONFIRMED | 356 | 99 |
| Labeled as FALSE POSITIVE | 77 | 738 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 1_DecisionTree
Decision Tree
- n_jobs: -1
- criterion: gini
- max_depth: 3
- explain_level: 2
Validation
- validation_type: split
- train_ratio: 0.75
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
7.0 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.377679 | nan |
| auc | 0.890856 | nan |
| f1 | 0.87619 | 0.357441 |
| accuracy | 0.846457 | 0.357441 |
| precision | 0.936752 | 0.943787 |
| recall | 1 | 0.0715385 |
| mcc | 0.677622 | 0.357441 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.377679 | nan |
| auc | 0.890856 | nan |
| f1 | 0.87619 | 0.357441 |
| accuracy | 0.846457 | 0.357441 |
| precision | 0.907895 | 0.357441 |
| recall | 0.846626 | 0.357441 |
| mcc | 0.677622 | 0.357441 |
Confusion matrix (at threshold=0.357441)
| Predicted as CONFIRMED | Predicted as FALSE POSITIVE | |
|---|---|---|
| Labeled as CONFIRMED | 385 | 70 |
| Labeled as FALSE POSITIVE | 125 | 690 |
Learning curves
Decision Tree
Tree #1
Rules
if (koi_count <= 1.5) and (koi_fwm_stat_sig <= 0.002) and (koi_smet <= 0.09) then class: FALSE POSITIVE (proba: 94.64%) | based on 1,679 samples
if (koi_count > 1.5) and (koi_dikco_msky <= 1.73) and (koi_prad <= 15.2) then class: CONFIRMED (proba: 92.05%) | based on 780 samples
if (koi_count <= 1.5) and (koi_fwm_stat_sig > 0.002) and (koi_dikco_msky <= 1.055) then class: CONFIRMED (proba: 64.26%) | based on 719 samples
if (koi_count <= 1.5) and (koi_fwm_stat_sig > 0.002) and (koi_dikco_msky > 1.055) then class: FALSE POSITIVE (proba: 93.98%) | based on 299 samples
if (koi_count <= 1.5) and (koi_fwm_stat_sig <= 0.002) and (koi_smet > 0.09) then class: FALSE POSITIVE (proba: 70.33%) | based on 209 samples
if (koi_count > 1.5) and (koi_dikco_msky > 1.73) and (koi_count <= 2.5) then class: FALSE POSITIVE (proba: 97.26%) | based on 73 samples
if (koi_count > 1.5) and (koi_dikco_msky <= 1.73) and (koi_prad > 15.2) then class: FALSE POSITIVE (proba: 94.12%) | based on 34 samples
if (koi_count > 1.5) and (koi_dikco_msky > 1.73) and (koi_count > 2.5) then class: CONFIRMED (proba: 62.5%) | based on 16 samples
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 4_Default_CatBoost
CatBoost
- n_jobs: -1
- learning_rate: 0.1
- depth: 6
- rsm: 1
- loss_function: Logloss
- eval_metric: Logloss
- explain_level: 2
Validation
- validation_type: split
- train_ratio: 0.75
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
2.3 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.0759057 | nan |
| auc | 0.996014 | nan |
| f1 | 0.980488 | 0.37257 |
| accuracy | 0.974803 | 0.37257 |
| precision | 1 | 0.995323 |
| recall | 1 | 4.09966e-05 |
| mcc | 0.945076 | 0.37257 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.0759057 | nan |
| auc | 0.996014 | nan |
| f1 | 0.980488 | 0.37257 |
| accuracy | 0.974803 | 0.37257 |
| precision | 0.974545 | 0.37257 |
| recall | 0.986503 | 0.37257 |
| mcc | 0.945076 | 0.37257 |
Confusion matrix (at threshold=0.37257)
| Predicted as CONFIRMED | Predicted as FALSE POSITIVE | |
|---|---|---|
| Labeled as CONFIRMED | 434 | 21 |
| Labeled as FALSE POSITIVE | 11 | 804 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 3_Default_Xgboost
Extreme Gradient Boosting (Xgboost)
- n_jobs: -1
- objective: binary:logistic
- eta: 0.075
- max_depth: 6
- min_child_weight: 1
- subsample: 1.0
- colsample_bytree: 1.0
- eval_metric: logloss
- explain_level: 2
Validation
- validation_type: split
- train_ratio: 0.75
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
1.8 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.0883173 | nan |
| auc | 0.994194 | nan |
| f1 | 0.979115 | 0.648877 |
| accuracy | 0.973228 | 0.648877 |
| precision | 1 | 0.997622 |
| recall | 1 | 0.000439883 |
| mcc | 0.94184 | 0.648877 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.0883173 | nan |
| auc | 0.994194 | nan |
| f1 | 0.979115 | 0.648877 |
| accuracy | 0.973228 | 0.648877 |
| precision | 0.98032 | 0.648877 |
| recall | 0.977914 | 0.648877 |
| mcc | 0.94184 | 0.648877 |
Confusion matrix (at threshold=0.648877)
| Predicted as CONFIRMED | Predicted as FALSE POSITIVE | |
|---|---|---|
| Labeled as CONFIRMED | 439 | 16 |
| Labeled as FALSE POSITIVE | 18 | 797 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 5_Default_RandomForest
Random Forest
- n_jobs: -1
- criterion: gini
- max_features: 0.9
- min_samples_split: 30
- max_depth: 4
- eval_metric_name: logloss
- explain_level: 2
Validation
- validation_type: split
- train_ratio: 0.75
- shuffle: True
- stratify: True
Optimized metric
logloss
Training time
3.0 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.234308 | nan |
| auc | 0.968137 | nan |
| f1 | 0.932313 | 0.313915 |
| accuracy | 0.909449 | 0.313915 |
| precision | 1 | 0.957824 |
| recall | 1 | 0.0491224 |
| mcc | 0.802268 | 0.313915 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.234308 | nan |
| auc | 0.968137 | nan |
| f1 | 0.932313 | 0.313915 |
| accuracy | 0.909449 | 0.313915 |
| precision | 0.895928 | 0.313915 |
| recall | 0.971779 | 0.313915 |
| mcc | 0.802268 | 0.313915 |
Confusion matrix (at threshold=0.313915)
| Predicted as CONFIRMED | Predicted as FALSE POSITIVE | |
|---|---|---|
| Labeled as CONFIRMED | 363 | 92 |
| Labeled as FALSE POSITIVE | 23 | 792 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
